1. Technical Field
The invention is related to a system and process for image-based relativistic rendering, and more particularly, to a system and process for transforming images to accurately simulate how a scene would be viewed by an observer that is traveling at any desired velocity up to the speed of light.
2. Related Art
In recent years, numerous image modeling and rendering techniques have been developed for generation of a large variety of special effects. Such techniques have been employed in areas such as manipulation or synthesis of single images, or of motion video or film sequences, and for generation of special effects for electronic games. However, currently available techniques do not have the capability to realistically display an image or scene as it would be viewed by an observer traveling at relativistic velocities. For example, in some science-fiction movies and computer games, a spacecraft or other object may move with a speed comparable to that of light, but nothing in its shape, brightness, color, or shadow shows the consequences of Einstein""s Special Theory of Relativity.
Previously, a special ray tracing technique [1] called REST has been used to describe relativistic effects in space-time on synthetic or artificial models. According to the REST-frame algorithm [2, 3, 4, 5], objects, light sources and cameras that may rest at different inertial reference frames are modeled. The tracing rays are transformed to different inertial reference frames. Intersection tests and light illumination calculations are then performed for the rays. As a result, a series of phenomena such as length contraction, time dilation [6] and Doppler shift effects [7, 8] have been visualized.
Further, a technique for shading of high-speed moving objects has also been used. This technique operates by transforming all the objects and light sources to the inertial reference frame of a camera [9]. However, as with the aforementioned REST technique, this shading technique also depends on synthetic or artificial models.
One problem with both the REST-frame technique and the shading technique is that both require ray tracing procedures and scene data structures that can only be applied to special synthetic models constructed in their own systems. Therefore, these existing techniques are not competent for transforming existing images to show relativistic effects over a range of velocities up to the speed of light. Another problem with these techniques is that the ray-tracing methods employed are computer intensive functions. In other words, they take a relatively large amount of computer power and time to render an image from the synthetic or artificial models employed. Further, because both techniques use artificial or synthetic models, the images that they produce tend to look artificial, and thus less photo-realistic than is typically desired.
Consequently, what is needed is a technique for transforming existing images to show relativistic effects without the need for synthetic or artificial models. Further, such a technique should be able transform an image with considerably less computer power than that required for the ray-tracing methods employed by the techniques described above. Finally, such a technique should be able to produce photo-realistic images.
It is noted that in the preceding paragraphs, the description refers to various individual publications identified by a numeric designator contained within a pair of brackets. For example, such a reference may be identified by reciting, xe2x80x9creference [1]xe2x80x9d or simply xe2x80x9c[1]xe2x80x9d. Multiple references are identified by a pair of brackets containing more than one designator, for example, [5, 6, 7]. A listing of the publications corresponding to each designator can be found at the end of the Detailed Description section.
The present invention involves a new system and process for image-based relativistic rendering of one or more single or sequential images of a scene. Such images may be produced by any conventional method, and include, for example, photographs, panoramas, motion video or film, or any other type of image captured by a camera or other image capture device. Relativistic rendering in accordance with the present invention transforms one or more initial images to produce photo-realistic images that accurately simulate how the scene would be viewed by an observer that is traveling at any desired velocity up to the speed of light. These rendered images are equal in resolution and quality to the initial images. Therefore, given a photo-realistic input image, a photo-realistic rendered image is generated. Such relativistic simulation is based on Einstein""s Special Theory of Relativity. Further, unlike previous techniques, relativistic rendering in accordance with the present invention does not depend upon artificial or synthetic models, or upon ray-tracing methods.
In fact, in stark contrast to the present invention, traditional methods of ray tracing for rendering images are based on a determination of very complex light interactions, with the ultimate goal of finding the color, shade, or intensity of each point in a view window. In general, ray tracing is based on the idea that reflection and refraction of light rays can be modeled by recursively following the path that light takes as it bounces through an environment. Traditionally, this path is determined from a point within the view window towards an object and then a light source that illuminates the object within the scene. Consequently, using basic principles of reflection and refraction, the color, shade, or intensity of pixels in a final image can be determined by tracing a plurality of light rays as they are reflected and refracted while traveling through a scene comprised of three dimensional models or objects. In other words, ray tracing is used to simulate the path that light rays take as they bounce around within the world, i.e. they are traced through a scene comprised of three-dimensional objects or models. The three-dimensional objects or models are either non-transparent, thus requiring a consideration of reflection of light rays, or partially or fully transparent, thus requiring a consideration of refraction of light rays.
When using traditional ray tracing techniques, if an object is reflective, a new reflected light ray is simply traced from the point of intersection with the object towards the direction of reflection. The reflected ray is the mirror image of the original ray, pointing away from the surface, with the angle of reflection equivalent to the angle of incidence. Further, if the object is to some extent transparent, then a refracted ray is traced into the surface of the object, with the light ray continuing through at least a portion of the object. If the materials on either side of the surface of the object have different indices of refraction, such as air on one side and water on the other, then the refracted ray will be bent to some degree. Expanding on these general principles, it is clear that the process of ray tracing becomes increasingly complex as multiple light sources are considered, and as multiple objects or models, having various degrees of transparency are used to populate a scene to be rendered using ray tracing. Consequently, ray tracing techniques tend to be computationally expensive.
In contrast, the relativistic rendering techniques of the present invention are not concerned with either the reflection or refraction of light rays, and further, as mentioned above, the techniques of the present invention do not require the use of artificial models or objects. Consequently, the computer power required for relativistic rendering in accordance with the present invention is considerably less than that required for the previous methods such as ray tracing.
In general, an image or picture of a scene actually records a group of light rays going through the optical center of the lens of a camera in a particular instant. For the purposes of this invention, each image pixel is considered to represent a single corresponding light ray. Thus, recording an image in any given initial inertial reference frame produces a set of rays relative to that initial reference frame. Each of these rays is then processed using the Lorentz transformation to synthesize a novel view from a new inertial reference frame. This new inertial reference frame may have any desired observational parameters. These parameters may include the observational speed and direction (i.e. the speed and direction of the new reference frame), as well as other conventional viewing parameters such as the angle and rotation of view, field of view and viewing aspect. Further, while the viewing parameters are unique to each reference frame, the origin from which the scene is viewed in each inertial reference frame is identical. In other words, the space-time point of the optical center of the lens in the initial inertial reference frame coincides with the point from which the scene is viewed in the new inertial reference frame. Consequently, choosing the light rays within a view port of an observer in the new reference frame results in a novel view of the scene, which has undergone relativistic transformations. As a result, photo-realistic synthesis of relativistic effects such as view deformation and Doppler shift of an image captured at conventional speeds using conventional methods is achieved by the present invention.
Specifically, one or more images captured in an initial inertial reference frame S, along with the parameters defining how each image was captured, are provided to an Image-Based Relativistic Rendering (IBRR) system. Parameters defining how each image was captured include information such as angular shift of the camera, field of view of the camera, and the aspect of the camera image relative to the origin of the S reference frame. In addition, viewing parameters chosen by a user are also provided to the IBRR system via a user interface. Further, this input may also be provided in other ways, such as, for example, use of a look-up table to provide parameters, random generation of parameters, computer generated parameters based on predetermined criteria, etc. These viewing parameters represent a virtual observer in a new inertial reference frame Sxe2x80x2, and include information such as relative velocity between the S and Sxe2x80x2 reference frames, the direction of the relative velocity, the angular shift of the virtual observer, the field of view of the virtual observer, and the aspect of the image as viewed by the virtual observer relative to a common origin between the S and Sxe2x80x2 reference frames. The IBRR system then transforms each image so that it appears as it would if the virtual observer in the Sxe2x80x2 reference frame had actually acquired the image while viewing the scene at any desired viewing angle, orientation and velocity.
For convenience, the coordinate systems in S and Sxe2x80x2 are preferably chosen so that their xyz- and xxe2x80x2yxe2x80x2zxe2x80x2-axes, respectively, are parallel, and their origins O and Oxe2x80x2, respectively, coincide with each other when the image is acquired by the camera at time t=txe2x80x2=0. However, non-parallel axes may also be used. In the event that non-parallel axes are used, conventional calculations are applied to convert the direction, orientation and origin of light rays in the S reference frame to their corresponding direction, orientation and origin relative to the Sxe2x80x2 reference frame.
In transforming an image, four coordinate systems are considered, two for each inertial reference frame. First, a global ground coordinate system relative to the scene being captured defines the initial inertial reference frame, i.e. the S reference frame. Second, a local camera coordinate system defines the orientation of the camera relative to the global ground coordinate system of the S reference frame. Third, a global observer coordinate system defines the new inertial reference frame, i.e. the Sxe2x80x2 reference frame. Fourth, a local observer coordinate system defines the orientation of the virtual observer within the global observer coordinate system of the Sxe2x80x2 reference frame.
Initially, to simplify subsequent calculations, the directions of both the camera and the virtual observer are oriented along the negative z-axis of the global ground coordinate system and the negative zxe2x80x2-axis of the global observer coordinate system, respectively. Consequently, the local coordinate systems of the camera and the observer coincide with the global ground coordinate system and the global observer coordinate system, respectively. However, alignment along any axis may be used; the negative z-axis is simply chosen for convenience. A new orientation, i.e. the local coordinate system, of the camera or the virtual observer is then specified by a group of angular shifts about the xyz- or xxe2x80x2yxe2x80x2zxe2x80x2-axes, respectively. The order of rotation, for example rotation about the x-axis, y-axis, then z-axis, or rotation about the z-axis, y-axis, then x-axis, etc. may also be used so long as the corresponding rotation matrixes described herein are properly constructed.
A unique unit 3D vector of the direction of an incoming light ray corresponding to each individual pixel in an observer""s view is determined using the row and column numbers of the pixel, the field of view, the width and height of the image in the observer""s view, and the point from which the scene is viewed. Based on the Lorentz velocity transformation, the unit vector of the direction of the light ray measured in the global reference frame of the ground is then determined. The direction of the incoming light ray, when viewed from the local coordinate system of the camera, is then computed. This direction is then used to determine the corresponding pixel in the original, untransformed, image by mapping the position of each of the pixels in the original image to the location in the Sxe2x80x2 reference frame where each of the light rays representing those pixels intersects a view port or viewing plane of the observer in the new reference frame. The end result is a relativistically transformed pixel-based image having the same resolution and quality as the original input image. However, when images are transformed to show the effect of relativistic velocities, an original rectangular image will map to an image having curved edges. Further, as the relativistic velocity increases, the deformation of the shape of the original image naturally increases. Consequently, in one embodiment the transformed image is cropped such that a user can observe a conventional rectangular image, or any other desired geometrical shape, representing a subsection of the transformed image. In a further embodiment, the size, and thus the effective field of view of the transformed image may be increased or decreased by zooming into or out of the image using conventional techniques.
In still another embodiment, the light ray, and thus the corresponding pixel, may be further transformed by determining the Doppler shift of the light ray by using a Doppler shift factor. The Doppler shift factor is related to the relative velocity between the observer and the camera. Next, the observed wavelength of the light ray, xcexxe2x80x2, is computed using the Doppler shift factor in combination with the wavelength of the light ray, xcex, from the original, untransformed, image. The wavelength xcex of the light ray is determined by conventional methods based on the color and intensity of the corresponding pixel in the original image.
In transforming a panorama, i.e. a cylindrical or spherical view of a scene obtained using conventional methods, two cases are considered. First, where the input image is a panorama, and it is desired to form a non-panoramic view of a portion of the original, untransformed, panorama from the view of an observer in the new inertial reference frame, the direction of each incoming light ray of the panorama is calculated as described above for the case of a single image. Again, as described above, the unit 3D vector of the direction of an incoming light ray corresponding to each individual pixel in the observer""s view is used to determine the corresponding pixels on the original, untransformed panorama. As discussed above, this transformed image may also be cropped so as to display a subsection of the transformed image in any desired shape. Further, this image may also be Doppler shifted as described above.
Second, where the input image is a panorama, and it is desired to transform the entire panorama to form a panoramic view in the new reference frame of the observer, the global ground coordinate system and the global observer coordinate system are assumed to coincide. Thus, all the pixels in the panorama are transformed. After the new panorama is generated, a user can interactively view the panorama using a conventional panorama player. Again, in either case, the Doppler shift of each of the light rays defining the transformed image or panorama may be determined as described above for the case of a single image.
In transforming a motion video or film produced from either a series of cameras, or from a moving camera, a determination is first made as to whether the motion video or film represents constant speed motion along a straight line, whether it represents constant speed motion along a curve, or whether it represents accelerated motion along either a straight line or a curve.
In general, in the first case where the motion video or film represents constant speed motion along a straight line, the video or film may be captured using either a moving camera, or a series of stationary cameras at specific points in space. Consequently, if a group of video or film cameras is arranged along a straight line, and capture a scene simultaneously, image frames selected from these cameras are processed using the IBRR system described above for single images to form a video or film sequence that a high-speed moving virtual observer would see. It should be noted that in this case, the frame rate of the observer is constant. Further, if the scene is static during the moment of capturing, i.e. all the frames in each stationary video camera are identical, only a single moving video or film camera may be used instead of a group of stationary video cameras. It should be noted that the speed of light is so high, that as the relative velocity between inertial reference frames increases, virtually every motion, and thus every scene captured in the initial inertial reference frame, is effectively static from the point of view of an observer moving at a speed approaching relativistic velocities.
For example, assuming use of a group of motion video or film cameras, the cameras are uniformly positioned along a straight line to capture a scene simultaneously. The separation distance between the cameras is determined by the desired frame rate of the video or film along with the camera velocity which is being simulated by use of a sequence of cameras. Further, it should be noted that this simulated camera velocity is not the relativistic velocity of the virtual observer which is obtained by transformation of the sequence of images represented by the motion video or film.
Similarly, assuming a single video or film camera is moving along the same straight line, the speed of the moving camera is set in such a way that the moving camera captures each frame at each position of the stationary cameras in turn. To achieve this, the required velocity of the moving camera is determined based on the desired frame rate and the effective distance between captured image frames. In this manner, a sequence of pictures captured by the moving camera is identical to a sequence of images consisting of a set of pictures extracted from a group of stationary cameras.
Thus, where a virtual observer moving along the same straight line with a desired velocity reaches the position where an image frame is captured at the moment that the stationary camera is capturing a frame of a video sequence, then the observer will view the same image frame. This remains true for each frame because each subsequent image frame is processed as if it were the first image frame. The space and time coordinates of the virtual observer, measured in Sxe2x80x2 are then calculated in accordance with the Lorentz Transformation as described above.
However, because the virtual observer is moving with a velocity relative to the camera, the moving virtual observer will see the images relatively earlier than the image frame captured by an ordinary moving camera. When moving at relativistic velocities, the time dilation effect described by the Special Theory of Relativity must be considered. The result is that an effective frame rate of viewing by the virtual observer may be calculated. However, because the frame rate of the virtual observer traveling at relativistic velocities would be extremely fast, the human eye could not view an image sequence displayed at such a frame rate. Therefore, this frame rate is preferably slowed to a much slower frame rate that the human eye is capable of observing, such as is used for conventional viewing of motion video or film. Consequently, the relativistically transformed image sequence is effectively viewed in slow motion.
In the second case, where the motion video or film represents constant speed motion along a curve, the motion of the virtual observer moving along the curve at a constant speed can be approximated by a group of constant speed motions along straight-line intervals. Again, the motion video or film may be captured using a group of stationary cameras along a curved path, or a single camera moving along the curved path with normal constant speed (if the scene is static during the moment of capturing, as described above). The frame rate of the virtual observer will still be constant since it only depends on the magnitude of the speed. The orientation of the video camera and the tangent direction of the curve at each image capture position is used in the processing of each image frame using the IBRR system described above for the case of a single image.
In the third case, although the treatment of accelerated motion along either a straight line or a curve properly belongs to General Relativity, such motion may still be roughly dealt with using Special Relativity as described above. Specifically, an accelerated motion of a virtual observer can be approximated by a group of motions along straight-line intervals. Consequently, a low-speed moving video or film camera having an acceleration proportional to that of a moving virtual observer can be used to capture the scene if the scene is static during the moment of capturing. Note in this case that the frame rate of the observer no longer keeps constant. However, the frame rate during video or film playback of the transformed images can simply be adjusted to compensate for this effect.
In addition to the just described benefits, other advantages of the present invention will become apparent from the detailed description which follows hereinafter when taken in conjunction with the accompanying drawing figures.